Fix issue with async_scheduling when dealing with chunked input #359
Conversation
🚧 CI BlockedThe main CI workflow was not started for the following reason:
|
Signed-off-by: Tianmu Li <[email protected]>
tianmu-li
force-pushed
the
async_scheduling_chunk_fix
branch
from
1f87a51 to
09c183d
Compare
🚧 CI BlockedThe main CI workflow was not started for the following reason:
|
tianmu-li
requested review from
mgawarkiewicz-intel,
piotrbocian and
wpyszka
as code owners
tianmu-li
changed the title
Signed-off-by: Tianmu Li <[email protected]>
Signed-off-by: Tianmu Li <[email protected]>
|
@afierka-intel Can you check if this fixes the issue with long context? It works from my tests. |
|
/run-gaudi-tests |
1 similar comment
|
/run-gaudi-tests |
|
/run-gaudi-tests |
✅ CI PassedAll checks passed successfully against the following vllm commit: |
|
/run-gaudi-tests |
|
/run-gaudi-tests |
✅ CI PassedAll checks passed successfully against the following vllm commit: |
|
/run-gaudi-tests |
✅ CI PassedAll checks passed successfully against the following vllm commit: |
|
/run-gaudi-tests |
Cherry-pick of #359 --------- Signed-off-by: Tianmu Li <[email protected]> Co-authored-by: Michał Kuligowski <[email protected]>
…-project#360) Cherry-pick of vllm-project#359 --------- Signed-off-by: Tianmu Li <[email protected]> Co-authored-by: Michał Kuligowski <[email protected]>
…-project#360) Cherry-pick of vllm-project#359 --------- Signed-off-by: Tianmu Li <[email protected]> Co-authored-by: Michał Kuligowski <[email protected]>
|
I propose to merge to 0.11 and main branches, and cherry-pick to 0.10 if no new issues detected. |
|
Will this change be merged into v0.10.2_next? |
|
/run-gaudi-tests |
|
Only codeowners and testowners can request to run Gaudi tests. Contact list: kzawora-intel, xuechendi, mswiniarsk, adobrzyn, mgawarkiewicz-intel, vivekgoe, afierka-intel, michalkuligowski, iboiko-habana, PatrykWo, kamil-kaczor, kfojcik-intel, ksmusz, wuxun-zhang, xuechendi, attafosu, ulivne, Kacper-Pietkun, iboiko-habana, jkaniecki |
|
@piotrbocian the fix is for blocker, Please approve. |
|
/run-gaudi-tests |
✅ CI PassedAll checks passed successfully against the following vllm commit: |
piotrbocian
merged commit 4a8c529
into
vllm-project:v0.10.2_next
…-project#360) Cherry-pick of vllm-project#359 --------- Signed-off-by: Tianmu Li <[email protected]> Co-authored-by: Michał Kuligowski <[email protected]>
8 participants
When dealing with chunked prompt (input sequence length > max_num_batched_tokens), sometimes there is no output token due to the chunked prompt, but the scheduler expects one. This PR addresses this issue and aligns the behavior with gpu_model_runner.